Pantomation: 


Scoring and Performance Interface 


by Tom DeWitt 


The Pantomation system is a video/computer in- 
strument which can read scores and follow body 
movement. Applied to visual music, the system 
allows notation similar to traditional music com- 
position and performance with control by gesture 
much like orchestral conducting. 


Pantomation scores are graphs of event-number in 
one dimension versus a parameter of change in 
the second dimension. The scores are prepared by 
hand on graph paper, much as a score is written 
on staff sheets for music. The format of the score 
is a bar graph sometimes called “Skyline” nota- 


tion or change in subtle parts of the reproduction 
can be accomplished in real time by “conducting” 
using a baton or the hands. The system is able to 
perceive the position and velocity of movement of 
the baton in three dimensions giving direct con- 
trol over three independent parameters. This is 
different from traditional conducting gestures 
which communicate by presenting a pattern of 
movement to the orchestra, but some simple pat- 
terns can be extracted by the Pantomation 
system. For example, the conductor may assign 
significance to a downward sweep or a horizontal 
sweep of the hand, and the computer interface 


interactive control). The camera's output signal is 
processed to isolate the colors used to specify in- 
formation. In video parlance this known as 
“chroma keying.” The decoded color is then 
located in space, and this information is sent to a 
computer which can process, record and output 
the information either in digital or analog form. 
Two implementations of the system are currently 
in use. A laboratory model which uses several 
large computers resides at the Image Processing 
Lab of RPI. Over thirty artists have visited the 
facility on a Services-to-the Field grant in Media 
Arts from the National Endowment. A portable 


Figure Two. Graphic notation. 2a. Matrix designating 
ten rows of inputs and ten rows of outputs. 2b. A se- 
quence of notes scored according to “skyline” notation. 
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Figure Four. The Electronic Pantograph interprets baton 
position to distribute a signal to four loudspeakers. 
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tion. The duration of each event in a score is 
separately scored along with the parameters of 
change, so a single event is notated using several 
graphs. To facilitate the composition process, 
many parameters can be notated on the same 
piece of graph paper using a color code for each 
parameter. For example, red lines can be used for 
notation of event duration, blue lines for change 
in pitch, green lines for change of timbre. In 
visual music created using video synthesis or com- 
puter graphics, these notations could be for 
change in image color, form, size, etc. 


Once a score is loaded in the Pantomation system 
it is played back exactly as it is written. Articula- 
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will effect a programmed change. However, the 
system normally is used as a non-contact three 
dimensional joystick. It can also be used as a non- 
contact keyboard or fretboard. One demonstra- 
tion of its use was the performance of an “invis- 
ible” bass fiddle which the performer plucked by 
moving a green glove. 


The Pantomation system is based on technology 
developed at the Image Processing Laboratory of 
Rensselaer Polytechnic Institute and the Electronic 
Music Studio of the State University of New York 
at Albany. A color television camera scans the 
graph (for reading notation) or the performer (for 


version of the system was recently constructed for 
an Apple II micro-computer. It was used to ex- 
hibit the system at the Electra Exposition at the 
Museum of the Modern Art of the City of Paris 
in 1984. 
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